Papers with pretrained masked language models
Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pretrained language models such as masked language model (MLM) have brought significant improvements to many NLU and NLG tasks. |
| Approach: | They propose a probabilistic masking scheme for the masked language model and a model with a uniform prior distribution on the masking ratio. |
| Outcome: | The proposed model outperforms BERT on a bunch of downstream NLG tasks. |
BARLE: Background-Aware Representation Learning for Background Shift Out-of-Distribution Detection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for OOD detection focus on identifying semantic-shift OOD samples . background-shift detection is more practical but challenging . |
| Approach: | They propose a background-aware representation learning approach for background-shift OOD detection in NLP. |
| Outcome: | The proposed method improves background-shift OOD detection while maintaining ID classification accuracy. |
Unsupervised Subtitle Segmentation with Masked Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods to improve subtitle segmentation are based on character counting and linguistically correct segmentation. |
| Approach: | They propose a method where subtitle breaks are predicted according to likelihood of punctuation . their approach is highly portable across languages and domains . |
| Outcome: | The proposed method obtained competitive results in terms of segmentation accuracy across metrics while also fully preserving the original text and complying with length constraints. |
Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views (2023.eacl-main)
Copied to clipboard
| Challenge: | Temporal concept drift is a problem of data changing over time. |
| Approach: | They benchmark 11 pretrained masked language models on a series of tests to evaluate temporal concept drift. |
| Outcome: | The proposed framework evaluates 11 pretrained masked language models on a series of tests . it aims to reveal how robust an MLM is over time and provide a signal in case it has become outdated . |
Controllable Paraphrase Generation for Semantic and Lexical Similarities (2024.lrec-main)
Copied to clipboard
| Challenge: | Lexically diverse paraphrases are crucial in data augmentation because they enhance the linguistic diversity of the corpus. |
| Approach: | They propose a controllable model for semantic and lexical similarities by attaching tags to the head of the input sentence. |
| Outcome: | The proposed model can paraphrase an input sentence according to the tags specified. |
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing pretrained language models for mental health detection are inadequate . one in four people worldwide suffers from mental disorders . |
| Approach: | They train and release two pretrained masked language models to benefit machine learning for mental healthcare research . they demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks. |
| Outcome: | The proposed models improve mental health detection tasks on several benchmarks and are available for free. |